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Abstract 

This  report  describes  the  approach  and  initial  results  of  a  systematic  investigation  of  individual 
and  team  C4ISR  communication  and  performance  in  complex  time-critical  targeting  scenarios 
over  a  sustained  period  of  time.  To  date,  there  have  been  few  systematic  and  experimental 
programs  of  research  on  the  effects  of  fatigue  on  complex  decisionmaking  (Mahan,  1992),  team 
communication,  coordination,  shared  awareness,  or  performance  (Mahan,  1994;.  Mahan,  Elliott, 
Dunwoody,  &  Marino,  1998).  In  this  report,  we  focus  our  efforts  on  aspects  of  C4ISR 
communication  and  coordination,  and  how  we  can  assess  impact  of  fatigue  on  complex  team 
communication  and  performance  over  time. 

Introduction 

USAF  command  and  control  (C2)  warfighters  face  increasingly  complex  enviromnents 
that  represent  the  essence  of  naturalistic  decisionmaking—  multiple  demands  for  enhanced 
vigilance,  rapid  situation  assessment,  and  coordinated  adaptive  response  (Cohen,  1993;  Klein, 
1993;  Mitchell  &  Beach,  1990;  Orasanu  &  Salas,  1991;  Orasunu  &  Connolly;  1993;  Rasmussen, 
1993).  In  tactical  C2  situations,  the  focus  is  on  dynamic  battle  management  and  time-critical 
targeting.  Information  updates  may  be  from  air  or  from  ground  sources.  Coordination  demand  is 
high—  reconnaissance  and  resource  allocation  depend  upon  close  coordination  between  ground 
and  air  forces  in  a  distributed  network  system  of  systems.  Situations  requiring  close 
coordination  and  adaptive  replanning  are  increasingly  prevalent  and  challenging. 

It  is  clear  that  challenges  within  these  battle  scenarios  are  critically  important  to  air  and 
ground  superiority.  Much  effort  has  been  focused  on  the  development  of  advanced  technology 
to  provide  and  represent  time-critical  information  during  mission  execution.  These  capabilities 
are  needed  to  facilitate,  even  enable,  situation  awareness  and  coordinated  response  in  conditions 
of  information  complexity  and  time  pressure.  However,  technology  can  only  support,  not 
replace,  the  role  of  the  war  fighters.  In  fact,  technology  increases  the  role  and  demands  of  the 
human  decision  maker. 

Soldiers  and  leaders  make  decisions  and  responses  in  time-critical  situations.  While 
advanced  technology  affords  paradigm  shifts  in  information  technology,  it  cannot  replace  C2 
decision  makers  or  troops  on  the  ground,  who  must  make  tactical  decisions  under  duress,  often 
for  long  periods  of  time.  Despite  any  particular  advanced  technology,  individual  perfonnance 


will  still  vary,  depending  on  the  competence  of  each  individual  with  regard  to  situation  demands. 
We  need  to  determine  how  to  enhance  the  processes  by  which  war  fighters  recognize,  interpret 
and  respond  effectively  in  these  situations. 

Soldiers  and  leaders  make  decisions  and  responses  in  team  situations.  In  these 
situations,  effective  response  inevitably  involves  interaction  with  others:  typically,  information 
and/or  resources  must  be  distributed  effectively  and  actions/events  must  be  sequenced.  Current 
military  scenarios  comprise  operators  and  technology  in  a  complex,  dynamic,  and  interdependent 
system.  Complex  coordination  must  include  adaptive  problem  solving.  Prime  examples  include 
dynamic  re-allocation  of  assets,  for  purposes  such  as  retargeting  and  search-and-rescue. 

Soldiers  and  leaders  make  decisions  and  responses  in  sustained  situations.  Sustained 
operations  are  integral  to  command  and  control — combat  missions  require  vigilance  over  time 
and  adaptive  perfonnance  under  stress.  During  the  early  stages  of  actual  scenarios,  members  of 
the  command  center  are  often  up  for  several  days  with  little  if  any  time  for  recuperative  sleep. 
Over  time,  chronic  fatigue  will  affect  everyone,  and  the  likelihood  of  error  will  increase  (Hursh, 
1998).  This  is  particularly  relevant  to  C2  situations,  which  require  constant  monitoring,  even 
when  events  are  still. 

Sustained  operations  result  in  acute  and  chronic  fatigue  effects.  There  is  extensive 
documentation  on  the  negative  impact  of  acute  and/or  chronic  sleep  loss.  In  a  review  of  findings, 
Bonnett  (2000)  report  an  array  of  negative  effects.  These  effects  include  mood  changes, 
disorientation,  irritability,  perceptual  distortions,  hallucinations,  difficulty  in  concentration, 
and/or  paranoid  thinking,  depending  on  the  extent  of  sleep  loss.  Negative  effects  have  also  been 
demonstrated  on  a  range  of  cognitive  tests,  such  as  monitoring  tasks,  speed/accuracy  tests,  short¬ 
term  memory,  logical  reasoning,  and  mental  subtraction/addition.  Physiological  effects  are 
reflected  in  a  variety  of  physiological  tests,  such  as  EEG,  nystagmus,  hand  tremor,  slurring  of 
speech,  sluggish  corneal  reflexes,  hyperactive  gag  reflex,  and  increased  sensitivity  to  pain. 

The  effects  of  fatigue  on  complex  team  communication  and  performance  have  not 
been  extensively  investigated.  While  extensive  data  are  available  on  effects  of  sleep  loss  on 
physiological,  attitudinal,  and  cognitive  function  (Kryger,  Roth,  &  Demnet,  2000),  very  few 
studies  reported  data  regarding  sleep  loss  effects  on  particular  aspects  of  information  processing 
in  complex  decision  making  tasks  (Mahan,  1992;  1994)..  Even  fewer  have  reported  on  effects  on 
team  performance;  however,  a  few  prelimary  studies,  based  on  team  simulation-based 


performance,  provide  some  introductory  results  (Mahan,  et  al.  ,1998;  Coovert  et  al.,  2001).  To 
continue  this  stream  of  research,  the  Chronobiology  and  Sleep  Laboratory  at  Brooks  City-Base, 
San  Antonio,  TX  has  initiated  a  program  of  research  on  effects  of  sleep  loss  on  information 
processing,  communication,  coordination,  and  decision  making  in  complex  simulation-based 
tasks.  In  this  paper  we  shall  focus  on  fundamental  issues  related  to  team  communications 
(elicitation  and  assessment)  and  team  perfonnance. 

C4ISR  team  scenarios  were  chosen  for  USAF  operational  relevance  and  the  need  to 
understand  dynamics  of  communication,  shared  awareness,  coordinated  action,  and  adaptive 
response  to  time-critical  situations.  C4ISR  scenarios  were  crafted  such  that  mission  planning, 
team  coordination  and  dynamic  replanning  are  critical  to  team  success.  Scenarios  were  carefully 
constructed  to  ensure  equivalence  in  task  demand  and  difficulty  (Elliott,  Coovert,  &  Miller, 
2003).  We  predicted  that  fatigue  will  have  a  detrimental  effect  on  performance,  and  particularly 
on  indicators  of  problem  identification,  communication,  task  sequencing,  and  asset 
redistribution.  In  this  report,  we  focus  on  measures  of  communication  as  indicators  of 
information  exchange  and  task  sequencing,  and  describe  very  initial  relationships  with  other 
measures  of  team  process  and  perfonnance. 

Participants 

Research  participants  were  drawn  from  a  pool  of  USAF  officers  awaiting  Air  Battle 
Management  Training  at  Tyndall  AFB,  FL.  In  the  first  six,  three-person  teams,  there  was  only 
one  participant,  a  Captain,  who  was  not  a  Second  Lieutenant.  Three  of  the  first  six  teams  were 
composed  of  three  males,  while  the  other  three  teams  each  contained  two  males  and  one  female. 
All  of  the  participants  had  already  attended  the  Aerospace  Basics  Course,  which  however 
provided  them  with  little  training  or  knowledge  useful  for  the  current  study.  The  participants 
were  assigned  to  the  325th  Air  Control  Squadron,  Airborne  Battle  Managers  Undergraduate, 
Ground  (325  ACS-ABM-AUG). 

Training 

Each  participant  participated  in  a  40-hour  training  session  occurring  during  a  one-week  period. 
The  training  included  administrative  processing  ( 1  hr),  training  on  cognitive  test  battery  (9  hrs.) 
and  training  on  C4ISR  assets,  capabilities,  and  tactics,  along  with  AEDGE  interface  functions 


(30  hrs).  Also,  the  various  ergonomic  features  of  the  chairs  and  workstations  were  explained  and 
demonstrated  to  the  participants.  Participants  learned  principles  of  these  ergonomics  and  how 
they  can  be  adjusted  to  aid  with  fatigue  effects. 

Experimental  Manipulation 

The  experimental  session  began  at  6pm  on  the  last  day  of  training  (always  a  Friday)  and  ended  at 
1  lam  the  following  morning.  They  participated  as  three-person  teams,  every  other  hour,  in  8  40- 
minute  team-based  C4ISR  decision  making  scenarios,  with  20  additional  minutes  for  each 
session  for  debriefing,  data  collection,  and  mission  planning  for  the  next  session.  Every  other 
hour,  between  each  scenario  session,  they  perfonned  on  a  standard  cognitive  test  battery  that 
assesses  reaction  time,  working  memory,  and  multitasking.  They  also  provided  physiological 
(e.g.  temperature,  etc.),  mood-state,  and  sleepiness  data.  All  email  and  audio  communications 
were  digitally  captured  for  transcription.  This  resulted  in  extensive  cognitive  performance  and 
simulation-based  performance. 

Elicitation  of  Performance 

Criterion  measures  of  simulation-based  performance  were  generated  from  a  PC-based  synthetic 
team  task  environment  developed  for  investigations  of  C4ISR  team  perfonnance.  The  AWACS 
AEDGE  (Agent  Enabled  Decision  Group  Environment)  is  constructed  as  a  federation  of 
intelligent  agent-based  functions  that  enable  PC-based  scenario  construction  and  emulation  of 
C4ISR  information  and  cognitive  task  demands  (Hicks,  Stoyen,  Zhu,  2001).  These  PC-based 
scenarios  operate  much  like  networked  videogames.  However,  the  scenarios  are  scaled  (Elliott, 
Dalrymple,  Schiflett,  &  Miller,  in  press;  Schiflett  &  Elliott,  2000).  Functional  and  cognitive 
fidelity  was  based  on  cognitive  task  analyses  (Chaiken,  Elliott,  Dalrymple,  &  Schiflett,  2001; 
Elliott,  Dalrymple,  Regian,  &  Schiflett.,  2001). 

C4ISR  functions  in  these  scenarios  were  typified  by  a  strong  demand  for  communication, 
coordinated  action,  and  adaptive  response  to  time-critical  situations.  The  goals  were  to  (a) 
capture  operational  relevance  to  C4ISR  (content  fidelity),  (b)  identification  and  assessment  of 
individual  and  team  perfonnance  (construct  fidelity),  and  (c)  development  of  scenarios  that  are 
equivalent  in  difficulty  (cognitive  fidelity)  and  yet  (d)  are  not  exact  replicas  of  each  other.  If 
participants  perform  the  same  scenario  over  and  over,  over  time,  then  effects  of  fatigue  will  be 


confounded  with  effects  of  practice  and  recognition-based  perfonnance.  Scenarios  had  to  be 
demanding,  operationally  relevant,  equivalent  in  difficulty,  and  yet  be  distinct  from  each  other 
(Elliott  et  ah,  2003). 

The  C4ISR  functions  reflect  those  of  current  and  future  command  centers  that  coordinate 
across  diverse  functional,  geographic,  military,  national,  and  political  characteristics.  General 
principles  of  mission  goals  and  tactics  were  identified  and  reflected  in  the  AEDGE  Scenarios.  At 
the  same  time,  scenarios  must  reflect  research  goals — performance  constructs  must  be  identified, 
elicited,  and  assessed.  In  order  to  (a)  capture  current  and  future  operational  issues,  and  (b)  elicit 
interdependence,  coordinating  activities,  and  problemsolving  on  the  part  of  the  participants,  we 
chose  to  develop  C4ISR  scenarios  that  include  unplanned  events  requiring  coordination  among 
team  members,  problem  identification  and  problem  resolution. 

While  interface  functions  were  greatly  simplified,  careful  attention  was  given  to  capture 
of  core  information  processing  and  decisionmaking  demands  in  these  C4ISR  scenarios. 
Participants  were  trained  on  a  variety  of  enemy  and  friendly  weapons  platforms,  such  as  various 
UAV,  ISR,  fighter,  bomber,  SAM  and  other  missile  assets.  They  learned  typical  armament, 
speed,  and  radar  characteristics.  They  learned  the  relative  threat  and  designated  targets  of  each 
asset. 

Measures 

Mission  outcomes.  Raw  measures  of  mission  outcome  and  team  process  were  captured 
and  time-stamped  by  the  simulation.  This  includes  descriptions  and  counts  of  events  and  actions, 
which  then  form  the  basis  for  various  assessments  of  perfonnance.  For  example,  mission 
outcome  scores  were  represented  by  the  type,  number,  and  relative  value  of  assets  that  were  lost, 
by  “friendly”  and  “hostile”  roles.  Friendly  assets  included  air  bases,  cities,  surface-to-air  missile 
launchers,  uninhabited  aerial  vehicles,  tanker  aircraft,  high-value  reconnaissance  aircraft,  fighter 
aircraft,  and  bomber  aircraft.  Each  asset  was  given  a  relative  score  value,  generated  by  our 
weapons  director  expert,  and  validated  by  other  experienced  weapons  directors.  The  loss  of  any 
friendly  asset  detracts  from  the  score  of  the  friendly  team,  and  adds  to  the  score  of  the  enemy.  In 
turn,  hostile  assets  are  similar.  The  loss  of  hostile  assets  adds  to  the  score  of  the  friendly  team, 
and  detracts  from  the  score  of  the  hostile.  For  these  research  participants,  the  overall  mission 


outcome  score  was  based  on  the  point  value  obtained  after  subtracting  all  friendly  “losses”  from 
the  total  hostile  “losses”. 

Observation-based  measures.  These  measures  provide  additional  data  on  individual  and 
team  interaction.  Scenario  observation  sheets  provide  raters  with  descriptions  of  events  as  they 
evolve  in  the  scenario.  For  example,  in  the  first  6  minutes  each  scenario  introduces  several 
hostile  threats.  The  observation  sheets  also  describe  what  each  team  member  should  be  doing  at 
that  time.  During  the  first  6  minutes,  the  ISR  role  should  be  identifying  decoys,  to  enable 
STRIKE  to  jam  them  and  target  them  with  bombers.  At  the  same  time,  SWEEP  fighters  must 
protect  STRIKE  and  ISR  assets  from  air-to-air  threat.  Team  members  must  act  swiftly  and  in 
coordination  with  each  other,  or  they  will  likely  lose  assets  and  waste  resources. 
Behaviorally-anchored  rating  scales.  These  scales  are  based  on  descriptions  of  what  each 
person  should  be  doing  are  provided  for  time  segments  chosen  based  on  the  flow  of  events. 
Therefore,  the  appropriate  and  desirable  responses  to  these  events  are  also  similar,  across 
scenarios.  For  each  introduction  of  assets  and/or  targets,  the  appropriate  behaviors  were 
identified  for  each  role.  Descriptions  were  generated  for  different  levels  of  performance,  leading 
to  anchored  scales  for  observer-based  ratings  of  performance.  Figure  1  provides  an  example  of  a 
tailored  rating  scale,  this  one  tailored  to  rating  the  behavior  of  STRIKE,  during  the  first  six 
minutes  of  scenario  1 . 


0-6  min  OVERALL  rating  of  STRIKE  (Circle) 

During  this  time,  Strike  should  be  giving  orders  to  bombers  and  jammers,  jamming  SAM 
sites,  and  assisting  SWEEP  and  ISR. 

1  2  3  4  5 

POOR  FAIRADEQUATE  GOOD  EXCELLENT 

POOR:  no  actions  taken 

FAIR:  some  action,  very  ineffective 

ADEQUATE:  some  assigning  of  assets,  no  attention  or  assistance  to  others 
GOOD:  assets  assigned,  jamming  accomplished,  some  coordination  with  others 
EXCELLENT:  assets  assigned,  jamming  accomplished,  assistance  and  coordination  with  others 

Justify  poor  or  excellent  rating: 


Figure  1.  Observation-based  rating  scale  for  Strike,  Scenario  1,  1st  time  period 

Observation-based  ratings  of  team  communication.  For  specified  time  periods,  based 
on  events,  observers  provided  ratings  as  shown  in  Figure  2: 


0-6  min  OVERALL  rating  of  Team  Communications  (Circle) 

1  2  3  4  5 

POOR  FAIR  ADEQUATE  GOOD  EXCELLENT 

POOR:  No  comm  (or  inaccurate  comm)  on  conditions  of  friendlies  /  no  comm  on  coordination 
needs  or  activities 

FAIR:  Partial  comm  on  conditions  of  friendlies  /  hostiles;  some  comm  on  simple  coordination 
needs  /  activities 

ADEQUATE:  Some  comm  on  conditions  of  friendlies  /  hostiles;  some  comm  on  ordinary 
coordination  needs/activities 

GOOD:  Comm  on  conditions  of  friendlies/hostiles  covered  the  majority  of  the  situation;  comm 
on  more  complex  coordination  needs/activities 

EXCELLENT:  Comm  on  conditions  of  friendlies  covered  the  entire  situation;  comm  on 
conditions  of  hostiles  covered  the  entire  situation;  comm  on  the  most  complex  coordination 
needs;  comm  to  accomplish  the  most  complex  coordinations 

Justify  poor  or  excellent  rating: 


Figure  2.  Rating  scale  for  team  communications 


Audio  capture  of  communications .  Digitally  ecorded  communications  are  another 
critical  source  of  assessment.  Communications  were  initially  coded  for  indications  of  teamwork, 
such  as  sharing  of  information  or  assets,  sequencing  actions,  acknowledgements,  requests  for 
repeats,  task-related  encouragement,  expressions  of  fatigue,  and  social  comments  (positive  and 
negative).  All  comments  were  coded  as  to  whether  they  requested  or  provided  information. 

Analysis  of  communications  (computer-mediated  and  verbal).  In  addition,  all  audio  is 
digitally  captured,  and  will  be  coded  to  ascertain  the  degree  to  which  the  team  (a)  shared 
information  and  (b)  discussed  sequencing  their  activities,  for  each  phase  of  performance.  We 
plan  to  capture  the  number  of  times  each  team  member:  1)  requests  information  about  a  target, 
asset,  or  unknown  type  of  entity;  2)  provides  information  about  a  target,  asset,  or  unknown  type 
of  entity;  3)  engages  in  strategy  communications;  4)  requests  acknowledgements;  5)  provides 
acknowledgements;  6)  requests  repeats,  7)  provides  repeats;  8)  communicates  fatigue;  9) 
communicates  encouragement;  10)  communicates  on  the  time  remaining  in  the  scenario;  and  11) 
has  other  communications.  The  concept  map  tool  provides  a  useful  means  of  representing  these 
communications,  as  shown  in  Figure  3. 
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Figure  3.  Representation  of  communication  concepts 


The  first  two  types  above  deal  with  information  activities  (non-sequencing 
communications,  such  as  queries,  alerts,  reminders,  friendly  and  hostile  conditions,  lessons 
learned,  strategies  and  tactics,  etc.),  while  the  third  involves  coordinating  activities  (sequencing 
tasks,  sharing  assets,  requests  for  transfers,  orders  to  transfer,  coordination  of  transfers,  sending 
transfers,  receiving  transfers,  denying  transfers,  aborting  transfers,  etc).  We  will  then  correlate 
with  quantitative  indices  of  events,  such  as  the  number  of  times  a  handoff  request  or  "push" 
(when  a  teammember  offers  an  asset  to  another)  is  accepted  or  denied  (from  who  to  whom). 
Other  quantitative  indices  include  the  number  of  times  team  members  scope  in  or  out  (related  to 
SA),  the  number  of  times  they  open  information  windows  on  assets,  the  number  of  times  a  decoy 
is  correctly  identified,  assets  are  refueled  and/or  rearmed,  penetration  of  friendly  area  by  enemy; 
and  of  course,  all  outcome  measures  (targets  destroyed  by  whom,  etc).  Figure  4  represents  how 
patterns  of  communication  among  team  members  can  be  determined.  In  this  particular  example 
we  show  how  different  patterns  regarding  provision  and  request  of  information  (by  type)  can  be 
conceptualized.  In  this  case,  teams  that  have  high  amounts  of  providing  information  and  low 
amount  of  requests  are  demonstrating  implicit  coordination,  a  very  efficient  means  of 
communication  that  is  associated  with  expert  coordination  under  time  pressure. 


Figure  4.  Representation  of  communication  patterns  regarding  provision  and  request  of 
information  among  team  members. 


We  predict  these  indices  of  coordinating  behavior,  anticipatory  behavior,  and  SA-related 
behavior  will  decline  with  fatigue.  People  will  tunnel  in,  not  notice  the  predicaments  of  others, 
not  care  (share  resources),  and  not  realize  when  new  bandits  arise,  or  when  additional  friendly 
resources  are  available.  They  are  also  expected  to  fail  in  sequencing  activities,  such  that  decoys 
are  not  identified,  vulnerable  friendly  assets  are  lost,  and  hostile  targets  not  executed.  They  are 
also  expected  to  lose  friendly  assets  due  to  fuel  outs.  Data  collection  is  underway,  and 
preliminary  results  will  be  available  on  an  ongoing  basis. 

Discussion 

Data  collection  is  still  underway.  Coding  of  communications  have  resulted  in  acceptable 
reliability  (over  90%  agreement),  and  refinements  will  likely  increase  agreement.  Preliminary 
analyses  on  limited  data  have  already  revealed  significant  direct  effects  of  fatigue  on  some 
criterion  measures,  even  when  power  is  very  low  (N  =  4  teams,  within-subjects  time  series 
analyses).  Data  from  all  sources  will  be  ultimately  analyzed  using  a  multi-level  hierarchical 
modeling  approach  that  enables  consideration  of  performance  data  of  individuals,  clustered  in 
teams,  to  ascertain  relationships  and  trends  over  time  (Kreft,  &  De  Leeuw,  2000).  We  will  over 
time,  focus  on  how  individual  perfonnance  affects  team  performance,  and  how  both  are  affected 
by  fatigue. 
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